WIP initial steps for distributing compaction coordination#6217
Draft
keith-turner wants to merge 90 commits intoapache:mainfrom
Draft
WIP initial steps for distributing compaction coordination#6217keith-turner wants to merge 90 commits intoapache:mainfrom
keith-turner wants to merge 90 commits intoapache:mainfrom
Conversation
There is still only one coordinator, but now the TGW and compactors both could talk to multiple coordinators.
The set of shutting down tservers was causing system fate operations to have to run on the primary manager because this was an in memory set. This caused fate to have different code paths to user vs system fate, this in turn caused problems when trying to distribute compaction coordination. To fix this problem moved the set from an in memory set to a set in zookeeper. The set is managed by fate operations which simplifies the existing code. Only fate operations add and remove from the set and fate keys are used to ensure only one fate operation runs at a time for a tserver instance. The previous in memory set had a lot of code to try to keep it in sync with reality, that is all gone now. There were many bugs with this code in the past. After this change is made fate can be simplified in a follow on commit to remove all specialization for the primary manager. Also the monitor can now directly access this set instead of making an RPC to the manager, will open a follow on issue for this.
After this change meta fate and user fate are both treated mostly the same in the managers. One difference is in assignment, the entire meta fate range is assigned to a single manager. User fate is spread across all managers. But both are assigned out by the primary manager using the same RPCs now. The primary manager used to directly start a meta fate instance. Was able to remove the extension of FateEnv from the manager class in this change, that caused a ripple of test changes. But now there are no longer two different implementations of FateEnv
Before this change a fate client was only available on the primary manager. Now fate clients are avaiable on all managers. The primary manager publishes fate assignment locations in zookeeper. These locations are used by managers to send notifications to other managers when they seed a fate operation.
Contributor
Author
|
After merging in the changes from #6232 a basic test of compaction coordination spread across multiple managers is now working. Was failing before. Still alot of loose ends and refactoring that is needed, but the basic functionality seems to be working now. |
keith-turner
added a commit
to keith-turner/accumulo
that referenced
this pull request
Mar 23, 2026
Simplifies warn logging about inactive queues in the coordinator by using only information from the job queue to do the logging. Sets of compactors and information from the job queue was being used previously. Removes a usage of the running cache in the coordinator. Simplifying this logic to not use the running cache or sets of compactors will be helpful for apache#6217.
This logger was useful in 2.1. However in 4.0 its redundant with functionality in the monitor and is just extra code to maintain. Removing it also removes a usage of the running cache which may be helpful for apache#6217
Compaction and completion and failure were computing stats in the coordinator that needed the resource group and compactor address. This information was obtained from the running cache. Modified the RPCs to pass this information instead. This removes a usage of the running cache in coordinator which will be helpful for apache#6217.
This change is made in support of apache#6217. Successfully ran all ITs that were changed.
Modified code that used this RPC to get the same information directly from compactors instead. Making this change in support of apache#6217
Modified the command that calls this to instead reach out directly to the compactor. This change is made in support of apache#6217, it removes a useage of the running cahce and it simplifies the coordinators RPCs.
Contributor
Author
|
Did some manual testing of this branch with three managers. When the system was idle saw the managers using 300% cpu in top. Looked at thread view in top and saw threads w/ the name |
A test got stuck waiting for a table to go offline because of a dead compaction. Made the dead compaction detector run more frequently to handle this. Left some logging that was added to track this down.
Contributor
Author
Tracked down this problem, when an assistant manger was assigned an empty set of fate ranges it would use like 300% cpu. Made a quick fix in f4b8814 |
The same host+port in different resource groups in ZK would cause LiveTServerSet to continually add and remove the host port. Changed an internal map that keyed on host+port to instead key on the ZK path to fix this. fixes apache#6298
…RemovedCompactionStoreImpl.java Co-authored-by: Daniel Roberts <ddanielr@gmail.com>
…ction/coordinator/DeadCompactionDetector.java Co-authored-by: Daniel Roberts <ddanielr@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some initial steps for distributing compaction coordinator. Currently only contains the following.
Still need to do the following
Plan to continue experimenting with this.